Field-replaceable bank of immersion-cooled electronic components

ABSTRACT

A cooled electronic system and cooling method are provided, wherein a field-replaceable bank of electronic components is cooled by an apparatus which includes an enclosure at least partially surrounding and forming a compartment about the electronic components, a fluid disposed within the compartment, and a heat sink associated with the enclosure. The field-replaceable bank extends, in part, through the enclosure to facilitate operative docking of the electronic components into one or more respective receiving sockets of the electronic system. The electronic components of the field-replaceable bank are, at least partially, immersed within the fluid to facilitate immersion-cooling of the components, and the heat sink facilitates rejection of heat from the fluid disposed within the compartment. In one embodiment, multiple thermal conductors project from an inner surface of the enclosure into the compartment to facilitate transfer of heat from the fluid to the heat sink.

BACKGROUND

As is known, operating electronic components produce heat. This heatshould be removed in order to maintain device junction temperatureswithin desirable limits, with failure to remove heat effectivelyresulting in increased component temperatures, potentially leading tothermal runaway conditions. Several trends in the electronics industryhave combined to increase the importance of thermal management,including heat removal for electronic components, including technologieswhere thermal management has traditionally been less of a concern, suchas CMOS. In particular, the need for faster and more densely packedcircuits has had a direct impact on the importance of thermalmanagement. First, power dissipation, and therefore heat production,increases as device operating frequencies increase. Second, increasedoperating frequencies may be possible at lower device junctiontemperatures. Further, as more and more devices or components are packedonto a single chip, heat flux (Watts/cm²) increases, resulting in theneed to remove more power from a given size chip or module. These trendshave combined to create applications where it is no longer desirable toremove heat from modern devices solely by traditional air coolingmethods, such as by using air cooled heat sinks with heat pipes or vaporchambers. Such air cooling techniques are inherently limited in theirability to extract heat from electronic components with higher powerdensity.

The need to cool current and future high heat load, high heat fluxelectronic devices therefore mandates the development of aggressivethermal management techniques using, for instance, liquid cooling.

BRIEF SUMMARY

The shortcomings of the prior art are overcome, and additionaladvantages are provided through the provision of a cooled electronicsystem which includes, for instance, an electronic system comprising afield-replaceable bank of electronic components, and a cooling apparatusfacilitating cooling of the field-replaceable bank of electroniccomponents. The cooling apparatus includes an enclosure at leastpartially surrounding and forming a compartment about, at least in part,the electronic components of the field-replaceable bank of electroniccomponents, a fluid disposed within the compartment, and a heat sinkattached to or integrated with the enclosure. The field-replaceable bankof electronic components extend, in part, through the enclosure tofacilitate operative docking thereof into one or more respectivereceiving sockets of the electronic system. The electronic components ofthe field-replaceable bank of electronic components are, at leastpartially, immersed within the compartment in the fluid to facilitateimmersion-cooling thereof, and the heat sink facilitates rejection ofheat from the fluid disposed within the compartment.

In another aspect, a liquid-cooled electronics rack is provided whichincludes an electronics rack having at least one electronic system, theat least one electronic system including a field-replaceable bank ofelectronic components. A cooling apparatus is also provided whichfacilitates cooling of the field-replaceable bank of electroniccomponents. The cooling apparatus includes an enclosure at leastpartially surrounding and forming a compartment about, at least in part,the electronic components of the field-replaceable bank of electroniccomponents, a fluid disposed within the compartment, and a heat sinkattached to or integrated with the compartment. The field-replaceablebank of electronic components extends, in part, through the enclosure tofacilitate operative docking thereof into one or more respectivereceiving sockets of the at least one electronic system. The electroniccomponents of the field-replaceable bank of electronic components are,at least partially, immersed within the compartment in the fluid tofacilitate immersion-cooling thereof, and the heat sink facilitatesrejection of heat from the fluid disposed within the compartment.

In a further aspect, a method of facilitating cooling of an electronicsystem is provided. The method includes: providing an electronic systemcomprising a field-replaceable bank of electronic components; andproviding a cooling apparatus to facilitate cooling of thefield-replaceable bank of electronic components, the cooling apparatusincluding an enclosure at least partially surrounding and forming acompartment about, at least in part, the electronic components of thefield-replaceable bank of electronic components, a fluid disposed withinthe compartment, and a heat sink attached to or integrated with theenclosure. The field-replaceable bank of electronic components extend,in part, through the enclosure to facilitate operative docking thereofinto one or more respective receiving sockets of the electronic system.The electronic components of the field-replaceable bank of electroniccomponents are, at least partially, immersed within the compartment inthe fluid to facilitate immersion-cooling thereof, and the heat sinkfacilitates rejection of heat from the fluid disposed within thecompartment.

Additional features and advantages are realized through the techniquesof the present invention. Other embodiments and aspects of the inventionare described in detail herein and are considered a part of the claimedinvention.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

One or more aspects of the present invention are particularly pointedout and distinctly claimed as examples in the claims at the conclusionof the specification. The foregoing and other objects, features, andadvantages of the invention are apparent from the following detaileddescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1 depicts one embodiment of a conventional raised floor layout ofan air-cooled computer installation;

FIG. 2 is a front elevational view of one embodiment of an at leastpartially liquid-cooled electronics rack comprising multiple electronicsystems to be cooled via a cooling apparatus, in accordance with one ormore aspects of the present invention;

FIG. 3 is a schematic of an electronic system of an electronics rack andone approach to liquid-cooling of an electronic component with theelectronic system, wherein the electronic component is indirectlyliquid-cooled by system coolant provided by one or more modular coolingunits disposed within the electronics rack, in accordance with one ormore aspects of the present invention;

FIG. 4 is a schematic of one embodiment of a modular cooling unit for aliquid-cooled electronics rack such as illustrated in FIG. 2, inaccordance with one or more aspects of the present invention;

FIG. 5 is a plan view of one embodiment of an electronic system layoutillustrating an air and liquid-cooling approach for cooling electroniccomponents of the electronic system, in accordance with one or moreaspects of the present invention;

FIG. 6A is an elevational view of an alternate embodiment of aliquid-cooled electronics rack with immersion-cooling of electronicsystems thereof, in accordance with one or more aspects of the presentinvention;

FIG. 6B is a cross-sectional elevational view of one immersion-cooledelectronic system of the liquid-cooled electronics rack of FIG. 6A, inaccordance with one or more aspects of the present invention;

FIG. 7A is a cross-sectional elevational view of one embodiment of acooled electronic system including a field-replaceable bank ofelectronic components of an electronic system and a cooling apparatustherefor, taken along line 7A-7A in the plan view thereof of FIG. 7B, inaccordance with one or more aspects of the present invention;

FIG. 7B is a cross-sectional plan view of the cooled electronic systemof FIG. 7A, taken along line 7B-7B thereof, in accordance with one ormore aspects of the present invention;

FIG. 7C is a cross-sectional elevational view of a cooled electronicsystem comprising a field-replaceable bank of electronic components ofan electronic system and an alternate embodiment of a cooling apparatustherefor, in accordance with one or more aspects of the presentinvention;

FIG. 8 is a cross-sectional elevational view of another cooledelectronic system comprising a field-replaceable bank of electroniccomponents of an electronic system and a further embodiment of a coolingapparatus therefor, in accordance with one or more aspects of thepresent invention;

FIG. 9 is a top plan view of a further cooled electronic systemincluding a field-replaceable bank of electronic components of anelectronic system, and another embodiment of a cooling apparatustherefor, in accordance with one or more aspects of the presentinvention;

FIG. 10A is a top plan view of a still further cooled electronic systemcomprising a field-replaceable bank of electronic components of anelectronic system, and a cooling apparatus therefor, in accordance withone or more aspects of the present invention;

FIG. 10B is a partial side elevational view of the cooled electronicsystem of FIG. 10A, taken along line 10B-10B thereof, in accordance withone or more aspects of the present invention; and

FIG. 11 is a schematic of one embodiment of a cooled electronics rackcomprising one or more cooled electronic systems having one or morefield-replaceable banks of electronic components and associated coolingapparatuses, in accordance with one or more aspects of the presentinvention.

DETAILED DESCRIPTION

As used herein, the terms “electronics rack”, and “rack unit” are usedinterchangeably, and unless otherwise specified include any housing,frame, rack, compartment, blade server system, etc., having one or moreheat-generating components of a computer system, electronic system, orinformation technology equipment, and may be, for example, a stand alonecomputer processor having high-, mid- or low-end processing capability.In one embodiment, an electronics rack may comprise a portion of anelectronic system, a single electronic system, or multiple electronicsystems, for example, in one or more sub-housings, blades, books,drawers, nodes, compartments, etc., having one or more heat-generatingelectronic components disposed therein. An electronic system(s) withinan electronics rack may be movable or fixed, relative to the electronicsrack, with rack-mounted electronic drawers and blades of a blade centersystem being two examples of electronic systems (or subsystems) of anelectronics rack to be cooled.

“Electronic component” refers to any heat generating electroniccomponent of, for example, a computer system or other electronics unitrequiring cooling. By way of example, an electronic component maycomprise one or more integrated circuit dies and/or other electronicdevices to be cooled, including one or more processor dies, memory diesor memory support dies. As a further example, the electronic componentmay comprise one or more bare dies or one or more packaged dies disposedon a common carrier. Further, unless otherwise specified herein, theterms “liquid-cooled cold plate”, “liquid-cooled vapor condenser”,“liquid-cooled heat sink”, or “liquid-cooled thermal conductor” eachrefer to a thermally conductive structure having one or more channels orpassageways formed therein for flowing of liquid-coolant therethrough.

As used herein, a “liquid-to-liquid heat exchanger” may comprise, forexample, two or more coolant flow paths, formed of thermally conductivetubing (such as copper or other tubing) in thermal or mechanical contactwith each other. Size, configuration and construction of theliquid-to-liquid heat exchanger can vary without departing from thescope of the invention disclosed herein. Further, “data center” refersto a computer installation containing one or more electronics racks tobe cooled. As a specific example, a data center may include one or morerows of rack-mounted computing units, such as server units.

By way of further explanation, a “heat pipe” is a heat transfer devicewhich combines the principles of both thermal conductivity and phasetransition to effectively manage transfer of heat. A simple type of heatpipe includes a sealed case, an inner surface of which is covered with alayer of capillary or porous material, or structure comprising a wickwhich is saturated with the working fluid in its liquid phase. At a hotinterface within the heat pipe, which may be at a low pressure, aworking fluid within the heat pipe in contact with a thermallyconductive surface (for example, an inner wall of the casing or a wick),turns into a vapor by absorbing heat from that surface. The workingfluid vapor condenses back into a liquid at a cold interface of the heatpipe, releasing the latent heat. The working fluid liquid then returnsto the hot interface through, for example, the wick structure bycapillary action or gravity, where it evaporates once more and repeatsthe cycle. Internal pressure within the heat pipe can be set or adjustedto facilitate the phase change, depending on the demands of the workingconditions of the cooling system.

One example of the coolants discussed herein, such as the facilitycoolant or system coolant, is water. However, the concepts disclosedherein are readily adapted to use with other types of coolant on thefacility side and/or on the system side. For example, one or more ofthese coolants may comprise a brine, a dielectric liquid, a fluorocarbonliquid, a liquid metal, or other similar coolant, or a refrigerant,while still maintaining the advantages and unique features of thepresent invention.

Reference is made below to the drawings, which are not drawn to scale tofacilitate an understanding of the various aspects of the presentinvention, wherein the same reference numbers used throughout differentfigures designate the same or similar components.

As shown in FIG. 1, in a raised floor layout of an air-cooled datacenter 100 typical in the prior art, multiple electronics racks 110 aredisposed in one or more rows. A computer installation such as depictedin FIG. 1 may house several hundred, or even several thousandmicroprocessors. In the arrangement of FIG. 1, chilled air enters thecomputer room via floor vents from a supply air plenum 145 definedbetween the raised floor 140 and a base or sub-floor 165 of the room.Cooled air is taken in through louvered covers at air inlet sides 120 ofthe electronics racks and expelled through the backs, i.e., air outletsides 130, of the electronics racks. Each electronics rack 110 may haveone or more air-moving devices (e.g., fans or blowers) to provide forcedinlet-to-outlet air flow to cool the electronic components within thedrawer(s) of the rack. The supply air plenum 145 provides conditionedand cooled air to the air-inlet sides of the electronics racks viaperforated floor tiles 160 disposed in a “cold” aisle of the computerinstallation. The conditioned and cooled air is supplied to plenum 145by one or more air conditioning units 150, also disposed within datacenter 100. Room air is taken into each air conditioning unit 150 nearan upper portion thereof. This room air may comprise (in part) exhaustedair from the “hot” aisles of the computer installation defined byopposing air outlet sides 130 of the electronics racks 110.

FIG. 2 depicts one embodiment of a liquid-cooled electronics rack 200comprising a cooling apparatus. In one embodiment, liquid-cooledelectronics rack 200 comprises a plurality of electronic systems 210,which may be processor or server nodes (in one embodiment). A bulk powerassembly 220 is disposed at an upper portion of liquid-cooledelectronics rack 200, and two modular cooling units (MCUs) 230 arepositioned in a lower portion of the liquid-cooled electronics rack forproviding system coolant to the electronic systems. In the embodimentsdescribed herein, the system coolant is assumed to be water or anaqueous-based solution, by way of example only.

In addition to MCUs 230, the cooling apparatus depicted includes asystem coolant supply manifold 231, a system coolant return manifold232, and manifold-to-node fluid connect hoses 233 coupling systemcoolant supply manifold 231 to electronic systems 210 (for example, toliquid-cooled vapor condensers or liquid-cooled heat sinks (see FIGS.6A-11) disposed within the systems) and node-to-manifold fluid connecthoses 234 coupling the individual electronic systems 210 to systemcoolant return manifold 232. Each MCU 230 is in fluid communication withsystem coolant supply manifold 231 via a respective system coolantsupply hose 235, and each MCU 230 is in fluid communication with systemcoolant return manifold 232 via a respective system coolant return hose236.

Heat load of the electronic systems 210 is transferred from the systemcoolant to cooler facility coolant within the MCUs 230 provided viafacility coolant supply line 240 and facility coolant return line 241disposed, in the illustrated embodiment, in the space between raisedfloor 140 and base floor 165.

FIG. 3 schematically illustrates one cooling approach using the coolingapparatus of FIG. 2, wherein a liquid-cooled cold plate 300 is showncoupled to an electronic component 301 of an electronic system 210within the liquid-cooled electronics rack 200. Heat is removed fromelectronic component 301 via system coolant circulating via pump 320through liquid-cooled cold plate 300 within the system coolant loopdefined, in part, by liquid-to-liquid heat exchanger 321 of modularcooling unit 230, hoses 235, 236 and cold plate 300. The system coolantloop and modular cooling unit are designed to provide coolant of acontrolled temperature and pressure, as well as controlled chemistry andcleanliness to the electronic systems. Furthermore, the system coolantis physically separate from the less controlled facility coolant inlines 240, 241, to which heat is ultimately transferred.

FIG. 4 depicts one detailed embodiment of a modular cooling unit 230. Asshown in FIG. 4, modular cooling unit 230 includes a facility coolantloop, wherein building chilled, facility coolant is provided (via lines240, 241) and passed through a control valve 420 driven by a motor 425.Valve 420 determines an amount of facility coolant to be passed throughheat exchanger 321, with a portion of the facility coolant possiblybeing returned directly via a bypass orifice 435. The modular coolingunit further includes a system coolant loop with a reservoir tank 440from which system coolant is pumped, either by pump 450 or pump 451,into liquid-to-liquid heat exchanger 321 for conditioning and outputthereof, as cooled system coolant to the electronics rack to be cooled.Each modular cooling unit is coupled to the system supply manifold andsystem return manifold of the liquid-cooled electronics rack via thesystem coolant supply hose 235 and system coolant return hose 236,respectively.

FIG. 5 depicts another cooling approach, illustrating one embodiment ofan electronic system 210 component layout wherein one or more air movingdevices 511 provide forced air flow 515 in normal operating mode to coolmultiple electronic components 512 within electronic system 210. Coolair is taken in through a front 531 and exhausted out a back 533 of thedrawer. The multiple components to be cooled include multiple processormodules to which liquid-cooled cold plates 520 are coupled, as well asmultiple arrays of memory modules 530 (e.g., dual in-line memory modules(DIMMs)) and multiple rows of memory support modules 532 (e.g., DIMMcontrol modules) to which air-cooled heat sinks may be coupled. In theembodiment illustrated, memory modules 530 and the memory supportmodules 532 are partially arrayed near front 531 of electronic system210, and partially arrayed near back 533 of electronic system 210. Also,in the embodiment of FIG. 5, memory modules 530 and the memory supportmodules 532 are cooled by air flow 515 across the electronics system.

The illustrated cooling apparatus further includes multiplecoolant-carrying tubes connected to and in fluid communication withliquid-cooled cold plates 520. The coolant-carrying tubes comprise setsof coolant-carrying tubes, with each set including (for example) acoolant supply tube 540, a bridge tube 541 and a coolant return tube542. In this example, each set of tubes provides liquid-coolant to aseries-connected pair of cold plates 520 (coupled to a pair of processormodules). Coolant flows into a first cold plate of each pair via thecoolant supply tube 540 and from the first cold plate to a second coldplate of the pair via bridge tube or line 541, which may or may not bethermally conductive. From the second cold plate of the pair, coolant isreturned through the respective coolant return tube 542.

As computing demands continue to increase, heat dissipation requirementsof electronic components, such as microprocessors and memory modules,are also rising. This has motivated the development of the applicationof single-phase liquid-cooling solutions such as described above.Single-phase liquid-cooling, however, has some issues. Sensible heatingof the liquid as it flows along the cooling channels and acrosscomponents connected in series results in a temperature gradient. Tomaintain a more uniform temperature across the heat-generatingcomponent, the temperature change in the liquid needs to be minimized.This requires the liquid to be pumped at higher flow rates, consumingmore pump power, and thus leading to a less efficient system. Further,it is becoming increasingly challenging to cool all the heat sources ona server or electronic system using pumped liquid, due to the densityand number of components, such as controller chips, I/O components andmemory modules. The small spaces and number of components to be cooledmake liquid plumbing a complex design and fabrication problem andsignificantly raises the overall cost of the cooling solution.

Immersion-cooling is one possible solution to these issues. Inimmersion-cooling, the components to be cooled are immersed in adielectric fluid that dissipates heat through boiling. The vapor is thencondensed by a secondary, rack-level working (or system) fluid usingnode or module-level, finned condensers, as explained below.

Direct immersion-cooling of electronic components of an electronicsystem of the rack unit using dielectric fluid (e.g., a liquiddielectric coolant) advantageously avoids forced air cooling and enablestotal liquid-cooling of the electronics rack within the data center.Although indirect liquid-cooling, such as described above in connectionwith FIGS. 3 and 5, has certain advantages due to the low cost and wideavailability of water as a coolant, as well as its superior thermal andhydraulic properties, where possible and viable, the use of dielectricfluid immersion-cooling may offer several unique benefits.

For example, the use of a dielectric fluid that condenses at atemperature above typical outdoor ambient air temperature would enabledata center cooling architectures which do not require energy intensiverefrigeration chillers. Also, the use of liquid immersion-cooling may,in certain cases, allow for greater compaction of electronic componentsat the electronic subsystem level and/or electronic rack level sinceconductive cooling structures might be eliminated. Unlike corrosionsensitive water-cooled systems, chemically inert dielectric coolant(employed with an immersion-cooling approach such as described herein)would not mandate copper as the primary thermally conductive wettedmetal. Lower cost and lower mass aluminum structures could replacecopper structures wherever thermally viable, and the mixed wetted metalassemblies would not be vulnerable to galvanic corrosion, such as in thecase of a water-based cooling approach. For at least these potentialbenefits, dielectric fluid immersion-cooling of one or more electronicsystems (or portions of one or more electronic systems) of anelectronics rack may offer significant energy efficiency and higherperformance cooling benefits, compared with currently available hybridair and indirect water cooled systems.

In the examples discussed below, the dielectric fluid may comprise anyone of a variety of commercially available dielectric coolants. Forexample, any of the Fluorinert™ or Novec™ fluids manufactured by 3MCorporation (e.g., FC-72, FC-86, HFE-7000, and HFE-7200) could beemployed. Alternatively, a refrigerant such as R-134a or R-245fa may beemployed if desired.

FIG. 6A is a schematic of one embodiment of a liquid-cooled electronicsrack, generally denoted 600, employing immersion-cooling of electronicsystems, in accordance with an aspect of the present invention. Asshown, liquid-cooled electronics rack 600 includes an electronics frame601 containing a plurality of electronic systems 610 disposed, in theillustrated embodiment, horizontally so as to be stacked within therack. By way of example, each electronic system 610 may be a server unitof a rack-mounted plurality of server units. In addition, eachelectronic system includes multiple electronic components to be cooled,which in one embodiment, comprise multiple different types of electroniccomponents having different heights and/or shapes within the electronicsystem.

The cooling apparatus is shown to include one or more modular coolingunits (MCU) 620 disposed, by way of example, in a lower portion ofelectronics rack 601. Each modular cooling unit 620 may be similar tothe modular cooling unit depicted in FIG. 4, and described above. Themodular cooling unit includes, for example, a liquid-to-liquid heatexchanger for extracting heat from coolant flowing through a systemcoolant loop 630 of the cooling apparatus and dissipating heat within afacility coolant loop 619, comprising a facility coolant supply line 621and a facility coolant return line 622. As one example, facility coolantsupply and return lines 621, 622 couple modular cooling unit 620 to adata center facility coolant supply and return (not shown). Modularcooling unit 620 further includes an appropriately sized reservoir, pumpand optional filter for moving liquid-coolant under pressure throughsystem coolant loop 630. In one embodiment, system coolant loop 630includes a coolant supply manifold 631 and a coolant return manifold632, which are coupled to modular cooling unit 620 via, for example,flexible hoses. The flexible hoses allow the supply and return manifoldsto be mounted within, for example, a door of the electronics rackhingedly mounted to the front or back of the electronics rack. In oneexample, coolant supply manifold 631 and coolant return manifold 632each comprise an elongated rigid tube vertically mounted to theelectronics rack 601 or to a door of the electronics rack.

In the embodiment illustrated, coolant supply manifold 631 and coolantreturn manifold 632 are in fluid communication with respective coolantinlets 635 and coolant outlets 636 of individual sealed housings 640containing the electronic systems 610. Fluid communication between themanifolds and the sealed housings is established, for example, viaappropriately sized, flexible hoses 633, 634. In one embodiment, eachcoolant inlet 635 and coolant outlet 636 of a sealed housing is coupledto a respective liquid-cooled vapor condenser 650 disposed within thesealed housing 640. Heat removed from the electronic system 610 via therespective liquid-cooled vapor condenser 650 is transferred from thesystem coolant via the coolant return manifold 632 and modular coolingunit 620 to facility coolant loop 619. In one example, coolant passingthrough system coolant loop 630, and hence, coolant passing through therespective liquid-cooled vapor condensers 650 is water.

Note that, in general, fluidic coupling between the electronicsubsystems and coolant manifolds, as well as between the manifolds andthe modular cooling unit(s) can be established using suitable hoses,hose barb fittings and quick disconnect couplers. In the exampleillustrated, the vertically-oriented coolant supply and return manifolds631, 632 each include ports which facilitate fluid connection of therespective coolant inlets and outlets 635, 636 of the housings(containing the electronic subsystems) to the manifolds via the flexiblehoses 633, 634. Respective quick connect couplings may be employed tocouple the flexible hoses to the coolant inlets and coolant outlets ofthe sealed housings to allow for, for example, removal of a housing andelectronic subsystem from the electronics rack. The quick connectcouplings may be any one of various types of commercial availablecouplings, such as those available from Colder Products Co. of St. Paul,Minn., USA or Parker Hannifin of Cleveland, Ohio, USA.

One or more hermetically sealed electrical connectors 648 may also beprovided in each sealed housing 640, for example, at a back surfacethereof, for docking into a corresponding electrical plane of theelectronics rack in order to provide electrical and network connections649 to the electronic system disposed within the sealed housing when theelectronic system is operatively positioned within the sealed housingand the sealed housing is operatively positioned within the electronicsrack.

As illustrated in FIG. 6B, electronic system 610 comprises a pluralityof electronic components 642, 643 of different height and type on asubstrate 641, and is shown within sealed housing 640 with the pluralityof electronic components 642, 643 immersed within a dielectric fluid645. Sealed housing 640 is configured to at least partially surround andform a sealed compartment about the electronic system with the pluralityof electronic components 642, 643 disposed within the sealedcompartment. In an operational state, dielectric fluid 645 pools in theliquid state at the bottom of the sealed compartment and is ofsufficient volume to submerge the electronic components 642, 643. Theelectronic components 642, 643 dissipate varying amounts of power, whichcause the dielectric fluid to boil, releasing dielectric fluid vapor,which rises to the upper portion of the sealed compartment of thehousing.

The upper portion of sealed housing 640 is shown in FIG. 6B to includeliquid-cooled vapor condenser 650. Liquid-cooled vapor condenser 650 isa thermally conductive structure which includes a liquid-cooled baseplate 652, and a plurality of thermally conductive condenser fins 651extending therefrom in the upper portion of the sealed compartment. Aplenum structure 654 comprises part of liquid-cooled base plate 652, andfacilitates passage of system coolant through one or more channels inthe liquid-cooled base plate 652. In operation, the dielectric fluidvapor contacts the cool surfaces of the thermally conductive condenserfins and condenses back to liquid phase, dropping downwards towards thebottom of the sealed compartment.

System coolant supplied to the coolant inlet of the housing passesthrough the liquid-cooled base plate of the liquid-cooled vaporcondenser and cools the solid material of the condenser such thatcondenser fin surfaces that are exposed within the sealed compartment tothe dielectric fluid vapor (or the dielectric fluid itself) are wellbelow saturation temperature of the vapor. Thus, vapor in contact withthe cooler condenser fin surfaces will reject heat to these surfaces andcondense back to liquid form. Based on operating conditions of theliquid-cooled vapor condenser 650, the condensed liquid may be close intemperature to the vapor temperature or could be sub-cooled to a muchlower temperature.

Advantageously, in immersion-cooling such as depicted in FIGS. 6A & 6B,all of the components to be cooled are immersed in the dielectric fluid.The system fluid can tolerate a larger temperature rise, whilemaintaining component temperatures, thus allowing a smaller flow rate,and higher inlet temperatures, improving energy efficiency of theresultant cooling apparatus.

However, immersion-cooling of an electronic system, such as a server,may present problems with regards to servicing or replacement in thefield of one or more of the components of the immersion-cooledelectronic system. Servicing or replacing a component cooled via animmersion-cooled approach, such as described above in connection withFIGS. 6A & 6B, requires that the entire electronic system be drained,and that the sealed enclosure be opened to access the electroniccomponent(s) to be serviced or replaced. This can be a time consumingand costly procedure to perform, particularly at the customer's datacenter.

In accordance with the cooled electronic systems presented herein,examples of which are depicted in FIGS. 7A-11, a hybrid liquid-coolingapproach is disclosed, wherein the cooled electronic system includes anelectronic system having one or more field-replaceable banks (or sets)of electronic components, and one or more cooling apparatusesfacilitating cooling of the one or more field-replaceable banks ofelectronic components. Note that as used herein, a “bank” of electroniccomponents refers to a plurality of electronic components of the same ordifferent type arrayed in any pattern and grouped for operativeinsertion into or removal from an electronic system. The coolingapparatus includes an enclosure at least partially surrounding andforming a compartment about, at least in part, the electronic componentsof a respective field-replaceable bank of electronic components, a fluiddisposed within the compartment, and a heat sink attached or affixed toor integrated with the enclosure. The field-replaceable bank ofelectronic components extends, in part, out through the enclosure tofacilitate operative docking thereof into one or more respectivereceiving sockets of the electronic system. The electronic components ofthe field-replaceable bank of electronic components are, at leastpartially, immersed within the compartment in the fluid to facilitateimmersion-cooling thereof, and the heat sink facilitates rejection ofheat from the fluid disposed within the compartment. In oneimplementation, the fluid comprises a dielectric fluid, such as one ormore of the above-referenced dielectric fluids.

In one embodiment, one or more thermal conductors are disposed withinthe compartment of the enclosure and project from one or more innersurfaces or walls of the enclosure into the compartment to facilitatetransfer of heat from the fluid to the heat sink. For instance, the oneor more thermal conductors may comprise one or more heat pipes. In oneimplementation, multiple heat pipes project from at least one innersurface of the enclosure into the compartment to facilitate transfer ofheat from the fluid to the heat sink, and are interleaved within thecompartment with multiple electronic components of the field-replaceablebank of electronic components. For instance, the multiple heat pipes maybe in thermal communication at opposite ends thereof, with opposite,inner surfaces or walls of the enclosure to facilitate transfer of heatfrom the fluid within the compartment to the opposite, inner surfaces ofthe enclosure, and the heat sink may be a first heat sink, and thecooling apparatus include a second heat sink, with the first and secondheat sinks being in thermal communication with, and disposed close oradjacent to, different inner surfaces of the opposite, inner surfaces ofthe enclosure. In another embodiment, the multiple heat pipes mayproject downwards, into the compartment from an upper, inner surface ofthe enclosure, and the heat sink may be a liquid-cooled heat sinkincluding at least one coolant-carrying channel accommodating the flowof liquid coolant therethrough.

In one implementation, the heat sink may comprise a liquid-cooled heatsink having one or more coolant-carrying channels accommodating the flowof liquid coolant therethrough. Alternatively, the heat sink may be anair-cooled heat sink, with a plurality of air-cooled fins extendingtherefrom, and (for instance) one or more thermoelectric modules sizedand positioned to facilitate transfer of heat from a respective surfaceor wall of the at least one inner surface or wall of the enclosure tothe air-cooled heat sink. By way of specific example, thefield-replaceable bank of electronic components could comprise afield-replaceable bank of dual-in-line memory modules (DIMMs).

In a further embodiment, the heat sink may be a liquid-cooled heat sinkand the at least one thermal conductor may include at least oneliquid-cooled thermal conductor in fluid communication with theliquid-cooled heat sink. For instance, multiple liquid-cooled thermalconductors may extend into the compartment of the enclosure, forexample, interleaved within the compartment with multiple electroniccomponents of the field-replaceable bank of electronic components, andbe in fluid communication with coolant supply and return manifoldsdisposed (for instance) at opposite sides of the enclosure.

In further aspects described herein, the cooling apparatus may includeone or more compliant layers associated with the enclosure, forinstance, either as a layer within the compartment coupled to an upper,inner surface of the enclosure, and/or as the upper cover and/or base ofthe enclosure. The one or more compliant layers engage or are coupled tothe field-replaceable bank of electronic components when the coolingapparatus is operatively positioned to facilitate cooling thefield-replaceable bank of electronic components, for instance, engage aportion thereof, and thereby provide compliance to facilitate securedocking of the field-replaceable bank of electronic components into theone or more respective receiving sockets of the electronic system.

More particularly, and in one example only, the cooling apparatuses andcooling methods disclosed herein solve the issue of DIMM cooling andreplaceability by providing an immersion-cooled enclosure whichfacilitates cooling an entire field-replaceable DIMM bank via, forinstance, pool boiling of dielectric fluid within the enclosure andlocal vapor condensation. The cooled electronic system, comprising theelectronic system with the field-replaceable bank of electroniccomponents (e.g., DIMMs) and the cooling apparatus, is readily insertedinto and removed as a single unit from, for instance, multipleDIMM-receiving sockets of the electronic system. In this manner, as oneor more DIMMs within a bank of field-replaceable DIMMs fail, theimmersion-cooled DIMM bank may be removed as an entire unit and replacedas a whole, with the enclosure comprising the one or more failing DIMMsbeing returned to, for instance, a manufacturer for individual DIMMreplacement. This approach is particularly advantageous in cases whereseveral DIMMs within a field-replaceable bank of DIMMs are allowed tofail before replacement of the entire bank is required.

By way of example, multiple DIMMs may be installed as animmersion-cooled DIMM bank, where the DIMMs are cooled by pool boilingof an encapsulated dielectric fluid. In one embodiment, the substratesor boards of the bank of DIMMs extend out from the enclosure and areconfigured so that they may be plugged into a set of receiving sockets(or slots) on or in, for instance, a motherboard of the electronicsystem. In one implementation, the bank of DIMMs extend through a lower,compliant base of the enclosure, such as a polymeric plate, whichprovides compliance or flexibility to facilitate insertion of the bankof DIMMs in operative position within the receiving sockets of theelectronic system. In operation, vapor produced within the compartmentof the enclosure is advantageously condensed locally through contactwith one or more thermal conductors extending from one or more innersurfaces of the enclosure into the compartment and (in oneimplementation) interleaved with multiple electronic components of theDIMMs. That is, as one example, each thermal conductor is disposedbetween a respective pair of adjacent DIMMs. The thermally conductivesurfaces of the thermal conductors may be, in one implementation,surfaces of solid thermal conductors or surfaces of heat pipes, orsurfaces of hollow, liquid-filled conductors that are attached at (forinstance) the upper, inner surface of the enclosure, or at one or bothof two opposite, inner surfaces of the enclosure.

In one implementation, the sides of the enclosure may be fabricated of athermally conductive material, such as a metal, and the enclosure mayhave integrated therewith a liquid-cooled cold plate, or may havemultiple integrated, liquid-cooled cold plates. Alternatively, the heatsink could comprise an air-cooled heat sink which dissipates heat fromthe fluid within the compartment to air passing across the air-cooledheat sink. Heat transfer to such an air-cooled heat sink may befacilitated by one or more thermoelectric modules. The cover or upperplate of the enclosure could be made either of a compliant material,such as polymer, or a thermally conductive material, such as metal. Asnoted, the cover or upper plate could also be in thermal communicationwith a liquid-cooled cold plate, either integrated with the enclosure orseparable. In such an embodiment, the multiple thermal conductors wouldbe in thermal communication with, for instance, the upper surface of theenclosure, rather than (or in addition to) opposing sides of theenclosure. As vapor condenses, heat released is conducted by the thermalconductors to the associated heat sink(s). In one implementation, theheat released to the thermal conductors is conducted directly to liquidflowing through hollow thermal conductors disposed within thecompartment of the enclosure.

One possible approach to immersion-cooling DIMMs is to require the DIMMcooling enclosure to be sealed along the electronic system boardsurface, which would make field replacement of the DIMM(s) moredifficult. In contrast, immersion-cooling of individual DIMMs isimpractical and costly due to space limitation, and the number of DIMMsin a typical electronic system, such as a server. The cooled electronicsystems and cooling apparatuses disclosed herein solve these issues bypackaging the enclosure around, for instance, a set of DIMMs alone(generally referred to herein as a field-replaceable bank of electroniccomponents), making the entire DIMM bank readily replaceable at theend-user's data center. For servicing, the bank could then be sent to,for instance, a manufacturer, where the failed DIMM(s) could bereplaced, and the enclosure refilled and prepared for return tooperation, resulting in minimal chance for fluid loss at the data centerand improved service quality. The advantage of this approach is evengreater in cases where several DIMMs in a bank are allowed to failbefore a repair event is initiated, and the bank is returned to thefactory for repair and/or maintenance.

FIGS. 7A & 7B depict one example of a cooled electronic system,generally denoted 700, in accordance with one or more aspects of thepresent invention. Referring collectively to FIGS. 7A & 7B, cooledelectronic system 700 includes an electronic system 701 with a pluralityof receiving sockets 702 for operatively receiving a field-replaceablebank 710 of electronic components 711, which (in one example) maycomprise a set of dual-in-line memory modules (DIMMs). Each electroniccomponent 711 includes, in the depicted example, multiple components 712(such as memory chips or modules) disposed on opposite main sides of aboard or substrate 713. A cooling apparatus is provided for thefield-replaceable bank 710 of electronic components 711, which includesan enclosure 720 that at least partially surrounds and forms acompartment 730 about, at least in part, the electronic components 712of the bank 710 of field-replaceable components. As illustrated in FIG.7A, the substrates 713 of electronic components 711 of the bank offield-replaceable components extend out through, in part, enclosure 720,for instance, through respective openings in a base plate 722 ofenclosure 720. In one example, enclosure 720 includes base plate 722,cover plate 723, and one or more side walls 724, which may be sealed 725together via, for instance, an adhesive or epoxy material, to formfluid-tight compartment 730.

A potting material 721 is also provided in the base plate 722 openingsabout the substrates 713 extending through the base plate 722 ofenclosure 720. The openings in base plate 722 are appropriately sized tofacilitate projection of the substrates 713 from the enclosure andthereby facilitate operative docking thereof in the respective receivingsockets 702 of electronic system 701. In one example, base plate 722 isa flexible (or compliant) material, such as a polymeric material, whichprovides insertion compliance for the field-replaceable bank 710 ofelectronic components 711. Communication and electrical connectors 714associated with substrates 713 of the field-replaceable bank 710 ofelectronic components 711 facilitate operative coupling of theelectronic components 711 to the electronic system 701 with docking ofthe bank of components into the receiving sockets 702.

Cover or upper plate 723 may also be a flexible (or compliant) plate,such as a polymeric plate, which may include grooves or notches in theinner surface 727 thereof (i.e., the surface partially definingcompartment 730) that accommodate the electronic components 711, asillustrated in FIG. 7A. Nubs 726 may be provided in the upper surface ofupper plate 723 aligned to the respective electronic components 711 tofacilitate, for instance, asserting downward pressure on thefield-replaceable bank 710 of electronic components 711 to ensure goodelectrical, operative coupling of the field-replaceable bank ofelectronic components within the respective receiving sockets 702 ofelectronic system 701.

As illustrated, a fluid 731 substantially fills or partially fillscompartment 730, for instance, leaving a small vapor space 732 in theupper portion thereof. This fluid may comprise a dielectric fluid suchas described above, which surrounds the electronic components 711 withincompartment 730 of enclosure 720. In this manner, enclosure 720 is animmersion-cooling enclosure that is fully supported about, and coupledand sealed to the field-replaceable bank 710 of electronic components711.

Referring to FIG. 7B, in one implementation, the side walls 724 ofenclosure 720 are fabricated of a thermally conductive material, such asa metal, and a first heat sink 750 and a second heat sink 760 areprovided integrated with (or coupled to) opposing sides of theenclosure. In the example depicted, first heat sink 750 is a firstliquid-cooled heat sink, and second heat sink 760 is a secondliquid-cooled heat sink, each of which includes one or morecoolant-carrying channels 751, 761, respectively, which allow for theflow of liquid coolant therethrough. First liquid-cooled heat sink 750receives liquid coolant via an inlet coupling 752 and returns liquidcoolant via an outlet coupling 753. Similarly, second liquid-cooled heatsink 760 receives liquid coolant into one or more coolant-carryingchannels 761 via an inlet coupling 762 and returns coolant via an outletcoupling 763. The coolant inlet and outlet couplings 752, 753, 762, 763may each comprise, in one instance, quick connect couplings, such as thequick connect couplings referenced above. These quick connect couplingsfacilitate connection with, for instance, external, flexible tubingwhich has sufficient flexibility to facilitate insertion or removal ofthe field-replaceable bank of electronic components, with the coolingapparatus coupled thereto into the electronic system.

In one embodiment, the first liquid-cooled heat sink 750 is disposedadjacent to and in thermal communication with a first inner surface orwall 728 of enclosure 720, and the second liquid-cooled heat sink 760 isdisposed adjacent to and in thermal communication with a second innersurface or wall 729 of enclosure 720, which in the embodiment of FIGS.7A & 7B, are opposing, inner surfaces or walls of the enclosure. Heat isconducted through the thermally conductive sidewalls from the opposing,inner walls 728, 729 to the respective first and second liquid-cooledheat sinks 750, 760. As illustrated in FIG. 7B, the respective innerwalls 728, 729 of enclosure 720 may include recesses or notches 735sized and configured to receive therein opposite ends of thermalconductors 740. Note that the thermal conductors 740 may be, in oneexample, soldered into the recesses or notches 735 in the opposing,inner walls 728, 729 of enclosure 720 to facilitate good thermalconduction from the thermal conductors 740 into the opposing sidewalls,and hence to the first and second liquid-cooled heat sinks 750, 760.

In the depicted implementation, thermal conductors 740 may each comprisea solid thermally conductive material (e.g., a metal), or alternatively,may comprise heat pipes which facilitate transfer of heat from fluid 731within compartment 730 to the first and second liquid-cooled heat sinks750, 760 for dissipation to the auxiliary coolant, that is, the liquidcoolant passing through the liquid-cooled heat sinks, which in oneexample, may comprise water or an aqueous-based solution. Note also thatthe one or more coolant-carrying channels 751, 761 through therespective liquid-cooled heat sinks 750, 760 may comprise any desiredconfiguration, such as a single large coolant flow chamber, multipleparallel coolant flow channels, a single coolant flow channel of anydesired configuration, etc.

As noted, as a specific example, the field-replaceable bank 710 ofelectronic components 711 may comprise a bank of DIMMs which areimmersion-cooled using a cooling apparatus such as depicted in FIGS. 7A& 7B. In one embodiment, the enclosure could comprise metal on itssides, and a suitable dielectric-compatible polymer material, such asethylene-propylene-diene monomer (EPDM) rubber, as its base and coverplates. The upper polymer plate may be made with notches into which theDIMMs can slip into, as well as nubs over the DIMMs to provide theoperator with an indication of where to press down to ensure that theDIMMs are seated firmly into their respective receiving sockets of theelectronic system, for instance, on a printed circuit board. The baseplate may also be made of such a polymer material to allow additionalflexibility during installation of the DIMMs into their sockets.

In one fabrication approach, the DIMMs are inserted into the enclosurethrough the openings in the base polymer plate. Once inserted, the spacearound the DIMMs is potted, sealing the space between the DIMMs, theelectronic system circuit board, and the polymer base plate. The polymercover and base plates may also be sealed to the metal side wall(s)using, for instance, either epoxy alone, or epoxy in combination withmechanical attachments, such as mechanical fasteners. The enclosure ispartially filled with dielectric fluid, and the upper polymer plate isattached and sealed, defining the enclosure about the bank of electroniccomponents. In one implementation, two holes may be made in the coverplate 723, and dielectric vapor forced into compartment 730 to replacethe existing air. Once the air is removed, the holes may be sealed withepoxy or potting material.

In the depicted embodiment, the thermal conductors 740 are interleavedwith the electronic components 711. As noted, these thermal conductorsmay comprise heat pipes inserted into the spaces between adjacent DIMMs.The heat pipes act as condensers to locally condense any dielectricfluid vapor formed within the compartment due to dielectric fluidboiling from one or more surfaces of electronic components 711 beingcooled, or as sub-coolers to cool the fluid when not boiling. Heatreleased during condensation is conducted by the heat pipes (or thermalconductors) to the sides of the enclosure, and from there to theliquid-cooled heat sinks, through which water (or any suitable auxiliarycoolant) flows to carry away rejected heat from the heat pipes. Ratherthan have two separate inlet couplings and two separate coolant outletcouplings (as shown in FIG. 7B), the inlet couplings and the outletcouplings could be separately externally joined with tubing, leaving asingle coolant inlet coupling and a single coolant outlet coupling, eachcomprising a respective quick connect. Note that the use of quickconnects facilitates field-replaceability of the bank of electroniccomponents, allowing the assembly to be readily inserted into andremoved from the liquid coolant loop associated with the electronicsystem within which the bank of electronic components operatively docks.

FIG. 7C depicts an alternate embodiment 700′ of the cooled electronicsystem 700 of FIGS. 7A & 7B, wherein the cover or upper plate 723′ ofenclosure 720 comprises a rigid plate, such as a metal plate, foradditional stiffness, which allows the plate to be soldered at its edgesto the metal sidewall(s) 724 of enclosure 720. In this embodiment, aflexible layer 770 of material, such as a layer of polymeric material,may be provided on the upper, inner surface of enclosure 720, betweenthe electronic components 711 and upper plate 723′ of the enclosure 720to provide compliance to the electronic components and therebyfacilitate operative docking of the field-replaceable bank 710electronic components 711 within the respective receiving sockets 702 ofthe electronic system 701. Advantageously, the presence of flexiblelayer 770 reduces chances of damaging the electronic components 711(e.g., DIMMs) when an external force is applied to the enclosure 720during installation of the field-replaceable bank 710 of electroniccomponents into the receiving sockets 702.

FIG. 8 depicts a further variation, generally denoted 800, of the cooledelectronic system 700 of FIGS. 7A & 7B. In cooled electronic system 800,the upper plate 723 of enclosure 720 of FIGS. 7A & 7B is replaced with aliquid-cooled heat sink 810, that is, for instance, attached tosidewalls 724 as a cover or integrated therewith as a unitary structure.Liquid-cooled heat sink 810 is fabricated of thermally conductivematerial and includes one or more coolant-carrying channels 811 throughwhich the auxiliary coolant passes. Auxiliary coolant is received intoliquid-cooled heat sink 810 via a coolant inlet coupling 812 and isreturned via a coolant outlet coupling 813, which in one embodiment, maycomprise quick connect couplings such as those described above. Withliquid-cooled heat sink 810 disposed as (or as part of) the upper plateof the enclosure 720′, one or more layers of compliant material 820 maybe provided between the electronic components 711 and the upper, innersurface of the enclosure 720′ to provide compliance to the electroniccomponents 711 and thereby facilitate operative insertion of thefield-replaceable bank 710 of electronic components 711 into therespective receiving sockets 702 of the electronic system 701. As notedabove, this compliant material may comprise, by way of example, a layerof polymeric material. Notches or recesses 830 may be provided withinthe upper, inner surface of liquid-cooled heat sink 810 to accommodatethermal conductors 840, which as described above, extend intocompartment 730 of enclosure 720′, for instance, in between adjacentelectronic components 711 of the field-replaceable bank 710 ofelectronic components 711. In this configuration, the thermal conductors740 thus extend downward from the liquid-cooled heat sink 810 coupled toor integrated with enclosure 720′. In one implementation, the thermalconductors may be heat pipes, such as described above.

Note that the cooled electronic system 800 configuration of FIG. 8 wouldbe advantageous in cases where the length of the enclosure 720′ isconstrained. In one implementation, the thermal conductors 740 (e.g.,heat pipes) may be soldered to the upper liquid-cooled heat sink 810,rather than to the sidewalls 724 of the enclosure, as in the examples ofFIGS. 7A-7C. The design of FIG. 8 also provides shorter thermal transferpaths from the thermal conductors 840 to the liquid-cooled heat sink810, and potentially allows for an increase in the liquid-cooled heatsink surface area in contact with compartment 730. For example, the heatsink surface area exposed to the compartment could extend over theentire top of the enclosure. A smaller average thermal conductor toliquid-cooled heat sink heat transfer distance, and larger liquid-cooledheat sink coverage area, may result in an overall smallerconductor-to-auxiliary coolant thermal resistance compared, forinstance, to the sidewall-coupled, liquid-cooled heat sink(s) designs ofFIGS. 7A-7C.

FIG. 9 depicts another embodiment of a cooled electronic system,generally denoted 900, in accordance with one or more aspects of thepresent invention. This cooled electronic system 900 is similar tocooled electronic system 700 described above in connection with FIGS. 7A& 7B, but rather than having two separate liquid-cooled heat sinks 750,760 (see FIG. 7B) on opposing sides of the enclosure, in FIG. 9, theliquid-cooled heat sinks are configured as manifolds that arefluidically coupled via coolant-carrying thermal conductors 930, each ofwhich comprises one or more coolant-carrying channels. Thecoolant-carrying thermal conductors may, in one example, be brazed attheir edges 931 to the opposing enclosure walls 727, 728, and be influid communication with the coolant supply manifold 910 and coolantreturn manifold 920 at opposite sides of the enclosure. A coolant inletcoupling 940 and coolant outlet coupling 941 are provided to facilitateliquid coupling of the cooled electronic system to a coolant loopassociated with the electronic system within which the field-replaceablebank of electronic components 711 is to be operatively placed. Eachcoupling may be a quick coupling, such as described above, andfacilitate coupling the cooled electronic system 900 to, for instance,external tubing with sufficient flexibility to allow for the readyinsertion and removal of the field-replaceable bank of electroniccomponents with the cooling apparatus disposed about the electroniccomponents as illustrated. In one example, the auxiliary coolant flowingthrough the coolant supply manifold 910, the one or more channels of thethermal conductors 930, and the coolant return manifold 920, is water,or an aqueous-based solution. Note that the configuration of FIG. 9advantageously reduces the conductor-to-auxiliary coolant resistance bybringing the auxiliary coolant (e.g., condensing water) as close aspossible to the thermal conductor 930 surfaces, and thereby may help toimprove heat removal and condensation within the compartment 730.

FIGS. 10A & 10B depict a further variation of a cooled electronic system1000. In this embodiment, air-cooled heat sinks 1010 are disposed atopposite sides of the enclosure 720″ to facilitate rejection of heatextracted from compartment 730 to auxiliary airflow 1011 passing acrossthe air-cooled heat sinks 1010. One or more air-moving devices (notshown) may be associated with the electronic system to facilitate, inpart, airflow 1011 across the heat sink(s) 1010, and each air-cooledheat sink may comprise a plurality of parallel-extending, thermallyconductive fins 1012 which project outwards from, for instance, oppositesides of enclosure 720″, as illustrated in the partial view of FIG. 10B.In addition, in one implementation, one or more thermoelectric modules1020 may be coupled between the air-cooled heat sink(s) 1010 andenclosure 720″ to facilitate active transfer (i.e., electronic pumping)of heat towards the air-cooled heat sinks 1010 from the enclosure 720″.The one or more thermoelectric modules 1020 are electrically coupled1021 to, for instance, an electronics board of the electronic system,which provides power to the thermoelectric modules.

The thermoelectric modules or cooling elements operate electronically toproduce a cooling effect. By passing a direct current through the legsof a thermoelectric device, a heat flow is produced across the devicewhich may be contrary to that which would be expected from Fourier'slaw.

At one junction of the thermoelectric element, both holes and electronsmove away, towards the other junction, as a consequence of the currentflow through the junction. Holes move through the p-type material andelectrons through the n-type material. To compensate for this loss ofcharge carriers, additional electrons are raised from the valence bandto the conduction band to create new pairs of electrons and holes. Sinceenergy is required to do this, heat is absorbed at this junction.Conversely, as an electron drops into a hole at the other junction, itssurplus energy is released in the form of heat. This transfer of thermalenergy from the cold junction to the hot junction is known as thePeltier effect.

Use of the Peltier effect permits the surfaces attached to a heat sourceto be maintained at a temperature below that of a surface attached to aheat sink. What these thermoelectric modules provide is the ability tooperate the cold side below the ambient temperature of the coolingmedium (e.g., air or water). When direct current is passed through thethermoelectric modules, a temperature difference is produced with theresult that one side is relatively cooler than the other side. Thesethermoelectric modules are therefore seen to possess a hot side and acold side, and provide a mechanism for facilitating the transfer ofthermal energy from the cold side of the thermoelectric module to thehot side of the thermoelectric module.

By way of specific example, the thermoelectric modules 1020 may compriseTEC CP-2-127-06L modules, offered by Melcor Laird, of Cleveland, Ohio.

Note that each thermoelectric array may comprise any number ofthermoelectric modules, including one or more modules, and is dependent(in part) on the size of the electronic modules, as well as the amountof heat to be transferred.

The thermoelectric (TE) array may comprise a planar thermoelectric arraywith modules arranged in a square or rectangular array. Although thewiring is not shown, each thermoelectric module in a column may be wiredand supplied electric current (I) in series and the columns ofthermoelectric modules may be electrically wired in parallel so that thetotal current supplied would be I×sqrt(M) for a square array comprisingM thermoelectric modules, providing an appreciation of the inherentscalability of the array. In this way, if a single thermoelectric moduleshould fail, only one column is effected, and electric current to theremaining columns may be increased to compensate for the failure.

Table 1 provides an example of the scalability provided by a planarthermoelectric heat exchanger configuration such as described herein.

TABLE 1 Number of TE Modules (M) Heat Exchanger Size 81 585 mm × 585 mm(23.0 in. × 23.0 in.) 100 650 mm × 650 mm (25.6 in. × 25.6 in.) 121 715mm × 715 mm (28.2 in. × 28.2 in.) 144 780 mm × 780 mm (30.7 in. × 30.7in.) 169 845 mm × 845 mm (33.3 in. × 33.3 in.)

For a fixed electric current and temperature difference across thethermoelectric modules, the heat pumped by the thermoelectric array willscale with the number of thermoelectric modules in the plan area. Thus,the heat load capability of a 650 mm×650 mm thermoelectric heatexchanger will be 1.23 times that of a 585 mm×585 mm thermoelectric heatexchanger, and that of an 845 mm×845 mm will be 2.09 times greater. Notethat the size of the heat sink may need to grow to accommodate theincreased heat load.

FIG. 11 is a rack-level view of a hybrid liquid-cooled electronics rack,generally denoted 1100, employing selective immersion-cooling of certainelectronic components of one or more electronic systems in the rackunit, in accordance with one or more aspects of the present invention.By way of example, the hybrid liquid-cooled electronics rack 1100 mayinclude an electronics rack 1101 with a plurality of electronic systems1110 disposed, in the illustrated embodiment, horizontally so as to bestacked within the rack. As one example, each electronic system 1110 maybe a server unit of a rack-mounted plurality of server units. Inaddition, the electronic systems include multiple electronic componentsto be cooled, which (in one embodiment) comprise multiple differenttypes of electronic components having different cooling requirements.For instance, one or more cooled electronic systems 800 may be providedcomprising field-replaceable banks of immersion-cooled electroniccomponents, such as described above in connection with the embodiment ofFIG. 8, while other components, such as one or more processor modulesmay have liquid-cooled plates or immersion-cooled enclosures 1112coupled thereto, and multiple other components 1111 may be air-cooledonly via, for instance, an inlet-to-outlet airflow through electronicsrack 1101, provided by one or more air-moving devices (not shown).

The cooling apparatus of FIG. 11 is shown to include one or more modularcooling units (MCUs) 620 disposed, by way of example, in a lower portionof electronics rack 1101. Each modular cooling unit 620 may be similarto the modular cooling unit described above in connection with FIGS. 4,6A & 6B, and include, for example, a liquid-to-liquid heat exchanger forextracting heat from coolant flowing through the auxiliary (e.g.,system) coolant loop 630 of the cooling apparatus and dissipating heatwithin, for instance, a facility coolant loop 619, comprising a facilitycoolant supply line 621 and a facility coolant return line 622. As oneexample, facility coolant supply and return lines 621, 622 couplemodular coolant unit 620 to a data center facility coolant supply andreturn (not shown). Modular cooling unit 620 further includes anappropriately sized reservoir, pump, and optional filter for movingliquid-coolant under pressure through auxiliary or system coolant loop630. In one embodiment, system coolant loop 630 includes a coolantsupply manifold 631 and a coolant return manifold 632, which are coupledto modular cooling unit 620 via, for example, flexible hoses. Theflexible hoses allow the supply and return manifolds to be mountedwithin, for example, a door of the electronics rack hingedly mounted tothe front or back of the electronics rack. In one example, coolantsupply manifold 631 and coolant return manifold 632 each comprise anelongated, rigid tube vertically mounted to the electronics rack 1101,or to a door of the electronics rack.

In the embodiment illustrated, the coolant supply and return manifoldare in fluid communication with respective coolant inlet and coolantoutlet couplings 812, 813 of multiple electronic systems 1110 within therack. By way of example only, a single pass of coolant within electronicsystem 1110 may be provided. As noted above, fluidic coupling betweenthe electronic systems and coolant manifolds, as well as within theelectronic systems between the cooled electronic systems comprising thefield-replaceable banks of electronic components and the node-levelcoolant loop of the electronic system, can be via suitable quick connectcouplers.

In the specific example of FIG. 11, electronic system 1110 comprises aserver with four immersion-cooled subsystems including, for instance,two cooled electronic systems 800, comprising field-replaceable banks ofDIMMs, and two other components, such as processors which areimmersion-cooled via separate enclosures 1112. In this example, only thefield-replaceable banks of DIMMs are readily removable by disconnectingthe respective quick connects 812, 813 and removing thefield-replaceable banks from the electronic system 1110. Water, or anyother suitable auxiliary coolant, may be circulated at the rack level.As the coolant flows from one immersion-cooled enclosure to another,heat is discharged to the coolant, and the warm coolant is thenexhausted from the electronic systems for return to the modular coolingunit 620, where it is cooled via heat exchange with the chilled facilitycoolant, and then pumped back to the electronic systems.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprise” (andany form of comprise, such as “comprises” and “comprising”), “have” (andany form of have, such as “has” and “having”), “include” (and any formof include, such as “includes” and “including”), and “contain” (and anyform contain, such as “contains” and “containing”) are open-endedlinking verbs. As a result, a method or device that “comprises”, “has”,“includes” or “contains” one or more steps or elements possesses thoseone or more steps or elements, but is not limited to possessing onlythose one or more steps or elements. Likewise, a step of a method or anelement of a device that “comprises”, “has”, “includes” or “contains”one or more features possesses those one or more features, but is notlimited to possessing only those one or more features. Furthermore, adevice or structure that is configured in a certain way is configured inat least that way, but may also be configured in ways that are notlisted.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below, if any, areintended to include any structure, material, or act for performing thefunction in combination with other claimed elements as specificallyclaimed. The description of the present invention has been presented forpurposes of illustration and description, but is not intended to beexhaustive or limited to the invention in the form disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the invention.The embodiment was chosen and described in order to best explain theprinciples of one or more aspects of the invention and the practicalapplication, and to enable others of ordinary skill in the art tounderstand one or more aspects of the invention for various embodimentswith various modifications as are suited to the particular usecontemplated.

What is claimed is:
 1. A cooled electronic system comprising: anelectronic system comprising a field-replaceable bank of electroniccomponents; and a cooling apparatus facilitating cooling of thefield-replaceable bank of electronic components, the cooling apparatuscomprising: an enclosure at least partially surrounding and forming acompartment about, at least in part, the electronic components of thefield-replaceable bank of electronic components, and thefield-replaceable bank of electronic components extending, in part,through the enclosure to facilitate operative docking thereof into oneor more respective receiving sockets of the electronic system; a fluiddisposed within the compartment, the electronic components of thefield-replaceable bank of electronic components being, at leastpartially, immersed within the compartment in the fluid to facilitateimmersion-cooling thereof; and a heat sink attached to or integratedwith the enclosure, the heat sink facilitating rejection of heat fromthe fluid disposed within the compartment.
 2. The cooled electronicsystem of claim 1, further comprising at least one thermal conductor,the at least one thermal conductor being disposed within the compartmentof the enclosure and projecting from at least one inner surface of theenclosure into the compartment to facilitate transfer of heat from thefluid to the heat sink.
 3. The cooled electronic system of claim 2,wherein the at least one thermal conductor comprises at least one heatpipe, the at least one heat pipe facilitating transfer of heat from thefluid within the compartment to the heat sink attached to or integratedwith the enclosure.
 4. The cooled electronic system of claim 3, furthercomprising multiple heat pipes projecting from the at least one innersurface of the enclosure into the compartment and interleaved within thecompartment with multiple electronic components of the field-replaceablebank of electronic components.
 5. The cooled electronic system of claim4, wherein the multiple heat pipes are in thermal communication atopposite ends thereof, with opposite, inner surfaces of the enclosure tofacilitate transfer of heat from the fluid within the compartment to theopposite, inner surfaces of the enclosure.
 6. The cooled electronicsystem of claim 5, wherein the heat sink is a first heat sink and thecooling apparatus further comprises a second heat sink, the first heatsink and the second heat sink each being in thermal communication with,and disposed adjacent to, a different inner surface of the opposite,inner surfaces of the enclosure.
 7. The cooled electronic system ofclaim 4, wherein the heat sink comprises a liquid-cooled heat sinkcomprising at least one coolant-carrying channel accommodating the flowof liquid coolant therethrough.
 8. The cooled electronic system of claim4, wherein the heat sink comprises an air-cooled heat sink with aplurality of air-cooled fins extending therefrom, and at least onethermoelectric module sized and positioned to facilitate transfer ofheat from a respective inner surface of the at least one inner surfaceof the enclosure to the air-cooled heat sink.
 9. The cooled electronicsystem of claim 4, wherein the field-replaceable bank of electroniccomponents comprises a field-replaceable bank of dual-in-line memorymodules.
 10. The cooled electronic system of claim 4, wherein themultiple heat pipes project downward, into the compartment from anupper, inner surface of the enclosure, and the heat sink comprises aliquid-cooled heat sink comprising at least one coolant-carrying channelaccommodating the flow of liquid coolant therethrough.
 11. The cooledelectronic system of claim 2, wherein the heat sink comprises aliquid-cooled heat sink and the at least one thermal conductor comprisesat least one liquid-cooled thermal conductor in fluid communication withthe liquid-cooled heat sink.
 12. The cooled electronic system of claim11, further comprising multiple liquid-cooled thermal conductorsextending into the compartment of the enclosure and interleaved withinthe compartment with multiple electronic components of thefield-replaceable bank of electronic components.
 13. The cooledelectronic system of claim 1, wherein the cooling apparatus furthercomprises at least one compliant layer associated with the enclosure andcoupled to the field-replaceable bank of electronic components, the atleast one compliant layer providing compliance to facilitate securedocking of the field-replaceable bank of electronic components into theone or more respective receiving sockets of the electronic system.
 14. Acooled electronics rack comprising: an electronics rack comprising atleast one electronic system, the at least one electronic systemcomprising a field-replaceable bank of electronic components; and acooling apparatus facilitating cooling of the field-replaceable bank ofelectronic components, the cooling apparatus comprising: an enclosure atleast partially surrounding and forming a compartment about, at least inpart, the electronic components of the field-replaceable bank ofelectronic components, the field-replaceable bank of electroniccomponents extending, in part, through the enclosure to facilitateoperative docking thereof into one or more respective sockets of the atleast one electronic system of the electronics rack; a fluid disposedwithin the compartment, the electronic components of thefield-replaceable bank of electronic components being, at leastpartially, immersed within the compartment in the fluid to facilitateimmersion cooling thereof; and a heat sink attached to or integratedwith the enclosure, the heat sink facilitating rejection of heat fromthe fluid disposed within the compartment.
 15. The cooled electronicsrack of claim 14, further comprising multiple thermal conductors, themultiple thermal conductors being disposed within the compartment of theenclosure and projecting from at least one inner surface of theenclosure into the compartment to facilitate transfer of heat form thefluid to the heat sink, the multiple thermal conductors beinginterleaved within the compartment with multiple electronic componentsof the field-replaceable bank of electronic components.
 16. The cooledelectronics rack of claim 15, wherein the multiple thermal conductorsare in thermal communication at opposite ends thereof, with opposite,inner surfaces of the enclosure to facilitate transfer of heat from thefluid within the compartment to the opposite, inner surfaces of theenclosure, and wherein the heat sink is a first heat sink, and thecooling apparatus further comprises a second heat sink, the first heatsink and the second heat sink each being in thermal communication with,and disposed adjacent to, a different inner surface of the opposite,inner surfaces of the enclosure.
 17. The cooled electronics rack ofclaim 14, wherein the cooling apparatus further comprises at least onecompliant layer associated with the enclosure and coupled to thefield-replaceable bank of electronic components, the at least onecompliant layer providing compliance to facilitate secure docking of thefield-replaceable bank of electronic components into the one or morerespective receiving sockets of the electronic system.
 18. A methodcomprising: providing an electronic system comprising afield-replaceable bank of electronic components; and providing a coolingapparatus to facilitate cooling of the field-replaceable bank ofelectronic components, the cooling apparatus comprising: an enclosure atleast partially surrounding and forming a compartment about, at least inpart, the electronic components of the field-replaceable bank ofelectronic components, and the field-replaceable bank of electroniccomponents extending, in part, through the enclosure to facilitateoperative docking thereof into one or more respective receiving socketsof the electronic system; a fluid disposed within the compartment, theelectronic components of the field-replaceable bank of electroniccomponents being, at least partially, immersed within the compartment inthe fluid to facilitate immersion-cooling thereof; and a heat sinkattached to or integrated with the enclosure, the heat sink facilitatingrejection of heat from the fluid disposed within the compartment. 19.The method of claim 18, wherein providing the cooling apparatus furthercomprises providing the cooling apparatus with multiple thermalconductors, the multiple thermal conductors being disposed within thecompartment of the enclosure and projecting from at least one innersurface of the enclosure into the compartment to facilitate transfer ofheat from the fluid to the heat sink, the multiple thermal conductorsbeing interleaved within the compartment with multiple electroniccomponents of the field-replaceable bank of electronic components. 20.The method of claim 18, wherein providing the cooling apparatuscomprises providing the cooling apparatus with at least one compliantlayer associated with the enclosure and coupled to the field-replaceablebank of electronic components, the at least one compliant layerfacilitating docking of the field-replaceable bank of electroniccomponents into the one or more respective receiving sockets of theelectronic system.